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IMAGE PROCESSING METHOD AMD SYSTEM 
FOR INTERPOLATION OP RESOLUTION" 

BACKGROUND OF THE INVENTION 

5 1. Field of the Invention 

The present invention relates to an image processing 
■ technique for interpolation of resolution to enlarge input or 
original images. More particularly, the invention relates to an 
image processing method and an image processing system applicable 
10 to display controllers for various information processing systems, 
such as Liquid-Crystal Displays (LCDs) , Electroluminescent (EL) 
displays, and Plasma Display Panels (PDPs) ; and a computer program 
product having a computer readable medium and a computer program 
recorded thereon that performs the method. 
15 2. Description of the Related Art 

Conventionally, image data sent from computers and graphic 
or video data for ordinary Televisions (TVs) according to the NTSC 
(National Television System Standard Committee) system or for 
High-Definition TVs (HDTVs) have various resolutions. The 
20 available resolutions of these data are specified by their formats. 

when images with different resolutions are displayed, on the 
screen of a Cathode-Ray Tube (CRT) , quality deterioration of the 
displayed images can be avoided without any particular complicated 
process. This is because a CRT is equipped with the so-called 
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"multi-scan" function and thus, it can display all the images on 
the screen according to their resolution. 

UnliTce this, LCDs or PDPs are unable to have a function 
corresponding to the "multi-scan" function of CRTs, because they 
5 have a fixed number of pixels on their screen. Therefore, if the 
resolution of an input image is lower than the fixed, maximum 
resolution of a LCD or PDP, some interpolation is necessary to - 
enlarge the image to the whole screen. 

Particularly, in recent years, LCD panels with a very high 

10 resolution termed the "Ultra-Extended Graphics Array (UXGA) " for 
1600 x 1200 pixels have been developed and have begun being 
manufactured. From this point of view, when an input image has a 
less resolution than a display device, it is important how to enlarge 
the input image so as to accord with the fixed, maximum resolution 

15 of the display device. 

Formerly, to enlarge an input image for this purpose, a 
suitable interpolation process has been typically applied to the 
input image using a specific interpolation function on the basis 
of the well-known sampling theorem. 

20 " For example, with the "nearest neighbor" method, the value 

of a nearest neighboring pixel of a target pixel in an input image 
is used as the value of its interpolated pixel. With the "bilinear 
interpolation" method, which is a linear interpolation, a weighted 
average value of two or four pixels around a target pixel in an 
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input image is used as the value ox its interpolated pixel. With 
the "cubic convolution" method, a specific interpolation function 
(not shown here) is used as its approximation function to improve 
the interpolation accuracy compared with the nearest neighbor and 
5 bilinear interpolation methods. However, these three methods have 
the following disadvantages. 

Specifically, the nearest neighbor method preserves the 
steep edges of the regions and/or objects in the input image, where 
the "steep edges" are the edges having abrupt density change. 
Q 10 However, this method has a disadvantage that the image quality 
J 1 degrades due to j aggies formed at the boundaries or contours of 

*y the regions and/or objects. 

u The bilinear interpolation method suppresses the jaggies 

=pi occurring in the nearest neighbor method. However, the image 

I 15 quality degrades like the nearest neighbor method, because the edges 
* of the regions and/or objects in the image tend to blur due to the 

use of the weighted average values. 

The cubic convolution method suppresses or prevents the 
jaggies occurring in the nearest neighbor method like the bilinear 
2 0 interpolation method. At the same time as this, the cubic 
convolution method improves the sharpness or steepness of the edges 
of regions and/or objects in the image compared with the bilinear 
interpolation method. However, the interpolation function used in 
this method serves basically as a low-pass filter and as a result, 

-3- 
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there is a disadvantage that the edges of the regions and/or objects 
in the image tend to blur if the objects include characters or 
letters . 

As a device to suppress or prevent the j aggies at the 
5 contours and the blur? at the edges in the input image, the Japanese 
Non-Examined Patent Publication No- 8-171633 published in 1996 
discloses an image enlargement device. The basic configuration of 
this device is shown in Fig. 1. 

In Fig. 1, an input memory 101 stores the luminance or 
10 density value of every pixel of an input digital image and outputs 
the density value at a first read address specified by a controller 
105. 

An intermediate value detector 102 detects the maximum and 
minimum density values of the four input values sent from an output 
15 memory 104 and then, it outputs the two remaining values (i.e., 
intermediate values) thereof to an average value calculator 103. 

The average value calculator 103 calculates an average 
value of the two- intermediate values sent from the intermediate 
value detector 102 and then, it outputs the average value thus 
20 calculated to the output memory 104. 

The output memory 104 stores the value of the first read 
address outputted from the input memory 101 or the average value 
outputted from the average value calculator 103 to a write address 
specified by the controller 105 . When the controller 105 specifies 
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a second read address to the output memory 104, the output memory 
104 outputs the value at the specified second read address to the 
intermediate value detector 102. 

In operation, prior to image enlargement, the luminance or 
5 density values of the whole pixels of the input image are stored 
in the respective addresses of the input memory 101. Then, under 
the control of the controller 105, the value of the target pixel 
of the input image stored in the memory 101 is read out ' from the 
input memory 101 and then, the value thus read out is stored in 
10 output memory 104 as the value of a corresponding one of the 
interpolated pixels of an enlarged image. Next, the values of the 
four neighboring pixels around the target pixel (which are located 
at the corners of a square area and the target pixel is located 
at the center of the area), which have been already defined, are 
15 read out from the output memory 104. Then, they are sent to the 
intermediates value detector 102. 

In the detector 102, the maximum and minimum values of the 
four neighboring pixels are removed and then, the two remaining 
intermediate values thereof are sent to the average value calculator 
20 103. In the calculator 103 , the average value of the two remaining 
intermediate values is calculated and sent to the output memory 



104. 



In the output memory 104, the average value of the two 
remaining intermediate values thus sent is stored at the address 



'01 0.5/24 21: 04 FAX 03 3208 0644 



ARICHIKA&IZUMI 



-> BLUMENTHAL ( FOLEY |011 



of the corresponding one of the interpolated pixels of the enlarged 
image . 

However, the image enlargement device shown in Fig. 1 has 
the following problems: 
5 Although blurs for the oblique edges are improved, blurs 

for the horizontal and vertical edges are scarcely improved, which 
are approximately the same as those generated by the cubic 
convolution method. Also, almost no improvement is seen for thin 
lines having a width of one pixel. Moreover, the device 
10 configuration is not suitable to real-time processing and thus, 
the device is difficult to be mounted on an image processing 
apparatus or system. 

On the other hand, the Japanese Non-Examined Patent 
Publication No. 7-182503 published in 1995 discloses an image 
15 processing method and device, in which low-resolution image data 
including characters and lines and natural pictures is well 
converted to high-resolution image data. In this device, each of 
the pixels of low-resolution image data is enlarged to (N x M) pixels, 
thereby converting it to high-resolution image data, where N and 
20 M are integers greater than unity. 

The pixel value distribution evaluation circuit evaluates 
the distribution of the valves of neighboring pixels around a target 
pixel in the low-resolution image data. The representative value 
decision circuit decides two representative values from the values 
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of the neighboring pixels. The assignment circuit assigns the two 

representative values to the two regions formed by the separation 

circuit. Thus, the low-resolution image data is converted to the 

high-resolution image data in such a way that the characters and 

5 lines in the input image include clear edges and substantially no 

j aggies and at the same time, the natural pictures therein include 

substantially no blurs due to interpolation. 

However, the device disclosed by the Publication No. 

7-182503 is quite different in configuration as the invention, the 

~~ 10 reason of which will be clarified by the explanation presented 
SO 

£0 below, 

iy SUMMARY OF THE INVENTION 

& Accordingly, an object of the present invention is to 

£ 15 P rovid e an image processing method and an image processing system 
p that improve the quality of an enlarged image of an input or original 

image including characters and lines, and a computer program product 
that performs the method. 

Another object of the present invention is to provide an 
20 image processing method and an image processing system that make 
it possible to enlarge an input or original image in such a way 
' as to suppress or eliminate j aggies at the contours in the image, 
and a computer program product X-T^^X performs tile method. 

Still another object of the present invention is to provide 
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an image processing method and an image processing system that makes 
it possible to enlarge an input or original image in such a way 
as to suppress or eliminate j aggies at the contours and blurs at 
the edges, and a computer program product that performs the method. 
5 A further object of the present invention is to provide an 

image processing method and an image processing system that are 
applicable to real-time processing for enlarging an input or 
original image T and a computer program product that performs the 
method. 

10 The above objects together with others not specifically 

mentioned will become clear to those skilled in the art from the 
following description. 

According to a first aspect of the present invention, an 
image processing method is provided- This method comprises the 

15 steps of: 

(a) defining a local area in an input image; 

the local area including a target pixel and neighboring 
pixels surrounding the target pixel ; 

(b) applying a filtering operation to the target pixel and the 
20 neighboring pixels in the local area while all the pixels in the 

local area are successively assigned to the target pixel, thereby 
outputting a filtered image,- and 

(c) mixing the filtered image and the input image together at 
a specific mixing ratio, thereby forming an output image. 
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With the method according to the first aspect ox the present 
invention, in the step (b) , the filtering operation is applied to 
the target pixel and the neighboring pixels in the local area defined 
in the step (a) while all the pixels in the local area successively 
assigned to the target pixel. Thus, the filtered image is outputted. 
Then, in the steps <c} , the filtered image and the input image are 
mixed together at a specific mixing ratio, thereby forming the 
output image . 

As a result, the input image can be enlarged in such a way 
as to suppress or eliminate the j aggies at the contours; in other 
words, the enlarged image of the input image has smooth contours. 
This means that the quality of the enlarged image is improved. 

Also, since the input image is simply subjected to the 
filtering process in the step (b) and the mixing process in the 
step (c) , this method is applicable to real-time processing for 
enlarging the input image. 

In a preferred embodiment of the method according to the 
first aspect, in the step (b) of applying the filtering operation, 
the pixel having a median value of density values of all the pixels 
in the local area is extracted, and the median value thus extracted 
is used for forming the filtered image. 

In another preferred embodiment of the method according to 
the first aspect, in the step (b) of applying the filtering operation, 
(b-1) an average value of density values of all the pixels in the 
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local area is calwlated, (b-2) one of the pixels having a nearest 
density value to the average value in the local area is extracted,, 
and (b-3) the density value thus extracted is used for forming the 
filtered image - 

5 In still another preferred embodiment of the method 

according to the first aspect, a step of enlarging an original image 
at a specific enlarging ratio to form the input image is additionally 
provided. In this embodiment, it is preferred that the mixing ratio 
is adjusted according to the enlarging ratio. 
10 t According to a second aspect of the present invention, 

another image processing method is provided. This method comprises 
the steps of: 

(a) defining a local area in an input image; 

the local area including a target pixel and neighboring 
15 pixels surrounding the target pixel; 

(b) applying a filtering operation to the target pixel and the 
neighboring pixels in the local area while all the pixels in the 
local area successively assigned to the target pixel, thereby 
output ting a filtered image, - 

20 the filtered image having a jaggy different in phase from 

a 3^997 in the input image; and 

(c) mixing the filtered image and the input image together at 
a specific mixing ratio, thereby forming an output image; 

wherein the jaggy in the input image is suppressed by that 
-10- 
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in the filtered image in the step (c) , resulting in the output image. 

With the method according to the second aspect of the 
invention, the filtered image having the jaggy different in phase 
from the jaggy in the input image is output ted in the step (b) . 
5 Then, the filtered image and the input image are mixed together 
at a specific mixing ratio in the step (c) , thereby forming the 
output image. In the step (c) , the jaggy in the input image is 
suppressed by the jaggy in the filtered image to thereby form the 
output image. Accordingly, there are the same advantages as those 
10 in the method according to the first aspect . 

In a preferred embodiment of the method according to the 
second aspect, in the step (b) of applying the filtering operation, 
the pixel having a median value of density values of all the pixels 
. in the local area is extracted, and the median value thus extracted 
15 is used for forming the filtered image. 

In another preferred embodiment of the method according to 
the second aspect, in the step <b> of applying the filtering 
operation, (b-1) an average value of density values of all the pixels 
in the local area is calculated, (b-2) one of the pixels having 
20 a nearest density value to the average value in the local area is 
extracted, and (b-3) the density value thus extracted is used for 
forming the filtered image. 

In still another preferred embodiment of the method 
according to the second aspect, a step of enlarging an original 
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image at a specific enlarging ratio to form the input image is 
additionally provided. In this embodiment, it is preferred that 
the mixing ratio is adjusted according to the enlarging ratio - 
According to a third aspect of the present invention, an 
5 image processing system is provided. This system, which is used 
to perform the method of the first aspect, comprises : 
(a) means for defining a local area in an input image; 

the local area including a target pixel and neighboring 
pixels surrounding the target pixel; 
O io (b) a filter for applying a filtering operation to the target 

-J pixel and the neighboring pixels in the local area while all the 

'2 pixels in the local area are successively assigned to the target 

= " pixel, thereby outputting a filtered image; and 

Ijl '(c) an image mixer for mixing the filtered image and the input 

|I 15 image together at a specific mixing ratio, thereby forming an output 
M= image. 

With the system according to the third aspect of the 
invention, because of substantially the same reason as described 
in the method of the first aspect, there are the same advantages 
20 as those in the method of the first aspect. 

In a preferred embodiment of the system according to the 
third aspect, in the filter, the pixel having a median value of 
density values of all the pixels in the local area is extracted, 
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ima.ge - 

In another preferred embodiment of the system according to 
the third aspect, in the filter, an average value of density values 
of all the pixels in the local area is calculated, one of the pixels 
5 having a nearest density value to the median value in the local 
area is extracted, and the density value thus extracted is used 
for forming the filtered image. 

In still another preferred embodiment of the system 
according to the third aspect, an interpolation processor for 
10 enlarging an original image at a specific enlarging ratio through 
interpolation to form the input image is additionally provided. 
In this embodiment , the mixing ratio is determined according to 
the enlarging ratio based on interpolation performed in the 
interpolation processor. 
15 According to a fourth aspect of the present invention, 

another image processing system is provided. This system, which 
is used to perform the method of the second aspect, comprises: 
{a) means for defining a local area in an input image; 

the local area . including a target pixel and neighboring 
20 pixels surrounding the target pixel; 

(b) a filter for applying a filtering operation to the target 

pixel and the neighboring pixels in the local area while all the 
pixels in the local area successively assigned to the target pixel, 
thereby outputting a filtered image; 
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the filtered image having a jaggy different in phase from 
a jaggy in the input image; and 

(c) an image mixer for mixing the filtered image and the input 

image together at a specif ic mixing ratio, thereby forming an output 
5 image; 

wherein, the jaggy in the input image are suppressed by that 
in the filtered image in the step (c) , resulting in the output image. 

With the system according to the fourth aspect of the 
invention, because of substantially the same reason as described 
10 in the method of the second aspect, there are the same advantages 
as those in the method of the second aspect. 

In a preferred embodiment of the system according to the 
fourth aspect, in the filtering, the pixel having a median value 
of density values of all the pixels in the local area is extracted, 
IS and the median value thus extracted is used for forming the filtered 
image . 

In another preferred embodiment of the system according to 
the fourth aspect, in the filtering, an average value of density 
values of all the pixels in the local area is calculated, one of 
20 the pixels having a nearest density value to the average value in 
the local area is extracted, and the density value thus extracted 
is used for forming the filtered image. 

In still another preferred embodiment of the system 
according to the fourth aspect, a processor for enlarging an 
-14- 
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original image at a specific enlarging ratio to form the input image 
is additionally provided. In this embodiment, it is preferred that 
the mixing ratio is adjusted according to the enlarging ratio. 

According to a fifth aspect of the present invention, a 
5 display controlling apparatus is provided, which comprises the 
image processing system according to the third or fourth aspect 
of the invention. 

According to a sixth aspect of the present invention, a 
computer program product having a computer readable medium and a 
10 computer program recorded thereon is provided. The computer 
program is operable to generate an output image from an input image. 

This product, which performs the method according to the 
first aspect, comprises: 

(a) code that defines a local area in an input image; 

15 the local area including a target pixel and neighboring 

pixels surrounding the target pixel ; 

(b) code that applies a filtering operation to the target pixel 
and the neighboring pixels in the local area while all the pixels 
in the local area are successively assigned to the target pixel, 

20 thereby outputting a filtered image; and 

(c) code that mixes the filtered image and the input image 
together at a specific mixing ratio, thereby forming an output 
image . 

According to a seventh aspect of the present invention, 
-15- 
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another computer program product having a computer readable medium 
and a computer program recorded thereon is provided. The computer 
program is operable to generate an output image from an input image. 

This product, which performs the method according to the 
5 second aspect, comprises: 

(a) code that defines a local area in an input image; 

the local area including a target pixel and neighboring 
pixels surrounding the target pixel ; 

(b) code that applies a filtering operation to the target pixel 
10 and the neighboring pixels in the local area while all the pixels 

in the local area are successively assigned to the target pixel, 
thereby outputting a filtered image; 

the filtered image having j aggies different in phase from 
j aggies in the input image; and 
15 (c) code that mixes the filtered image and the input image 

together at a specific mixing ratio, thereby forming an Output 
image; 

wherein the jaggy in the input image is suppressed by that 
in the filtered image, resulting in the output image. 
20 In the systems according to the third and fourth aspects, 

as the means for defining a local area in an input image, any hardware 
or software may be used if it can provide this function. 



BRIEF DESCRIPTION OF THE DRAWINGS 
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In order that the present invention may be readily carried 
into effect, it will now be described with reference to the 
accompanying drawings . 

Fig. 1 is a functional block diagram showing the 
5 configuration of a prior-art image enlargement device. 

Fig. 2 is a, functional block diagram showing the 
configuration of an image processing system according to a first 
embodiment of the invention . 

Fig. 3A is a schematic diagram showing the process to an 
10 input image in the image processing system according to the first 
embodiment of Fig. 2. 

Fig. 3B is a schematic diagram showing the process of an 
input image by the median value extractor in the image processing 
system according to the first embodiment: of Fig. 2. 
15 Fig. 3C is a schematic diagram showing the process of 

another input image in the image processing system according to 
the first embodiment of Fig. 2. 

Fig- 4 is a functional block diagram showing the 
configuration of an image processing system according to a second 
2 0 embodiment of the invention. 

Fig. 5 is a functional block diagram showing the 
configuration of an image processing system according to a third 
embodiment of the invention. 

Fig . S is a schematic diagram showing the process to an input 
-17- 
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image in tins image processing system according to the third 
embodiment of Fig, 5. 

Fig. 7 is a functional block diagram showing the 
configuration of an image processing system according to a fourth 
5 embodiment of the invention. 

Fig. 8 is a schematic diagram showing the process to an input 
image in the image processing system according to the fourth 
embodiment of Fig. 7. 

Fig. 9a is a schematic diagram showing the interpolation 
10 function used in the conventional nearest neighbor method. 

Fig. 9B is a schematic diagram showing the interpolation 
function used in a conventional bilinear method. 

Fig. 9C is a schematic diagram showing the interpolation 
function used in the image processing system according to the fourth 
15 embodiment of Fig. 7, respectively. 

Fig. 10 is a functional block diagram showing the 
configuration of a display controlling device according to a fifth 
embodiment of the invention. 



20 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Preferred embodiments of the present invention will be 
described in detail below while referring to the drawings attached. 
FIRST EMBODIMENT 
An image processing system according to a first embodiment 
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of the invention has the configuration shown in Fig. 2, which is 
used to carry out an image processing method according to the first 
embodiment- 

Specifically, as shown in Fig. 2, the image processing 
5 system of the first embodiment comprises an image processor 2 for 
processing an input image i and for outputting the processed image 
as an output image 3 . The image processor 2 comprises a median value 
extractor 11, an image mixer 12 , and a mixing ratio setter 13. 

The median value extractor 11 provides a filtering 
10 operation that outputs the median value of the density or gray-scale 
levels of all the pixels (i.e., the pixel values) in a local area 
surrounding a target pixel of the input image 1. The median value 
is defined as a value in the middle of a series of the pixel values 
arranged in order of magnitude. 
15 The image mixer 12 mixes the image subjected to the 

filtering operation in the extractor 11 <i, e ,, the filtered image) 
and the input image 1 itself at a mixing ratio specified by the 
mixing ratio setter 13, generating a mixed image. The mixing 
operation of the mixer 12 is performed by pixel by pixel in whole 
20 the input image 1 . Then, the mixer 12 outputs the mixed or processed 
image as the output image 3. 

Next, the reason why the image processor 2 provides a 
smoothing {i.e., jaggy suppressing or eliminating) effect to the 
input image 1 is explained below with reference to Figs. 3A to 3C. 
-19- 
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Fig. 3A shows schematically the content of the filtering 
process of the processor 2. In Fig. 3, "1A" denotes an example of 
the input image 1, which includes 36 (= 6 x 6) pixels. "11A" denotes 
an intermediate image produced by applying the filtering function 
S of the median value extractor 11 to the input image 1A. denotes 
an example of the output image 3. "l : 1" in the mixer 12 means 
that the input image 1A and the intermediate image 11A (i.e., the 
output image of the extractor 11) are mixed together in the image 
mixer 12 at the mixing ratio of 1 : 1. 
10 A s seen from Fig. 3A, to make the explanation simpler, the 

input image 1A includes only black and white pixels here. Also, 
the input image 1A has a jaggy at its oblique contour or boundary 
of a region or an object. When the input image 1A is subjected to 
the filtering operation of the median value extractor 11, the 
15 intermediate image 11A is generated. 

The filtering function of the median value extractor 11 is 
carried out in the following way. 

Fig. 3B shows an example of the filtering operation of the 
extractor 11. In this example, the ^local area- for median value 
20 extraction is a square area including nine pixels in total, where 
the target pixel is located at the center of the square area and 
the remaining eight pixels are located to surround the target pixel. 
Thus, this square local area is used for the filtering operation 
that extracts the median value. So, the median value extraction 
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filter using the "square local area" described here may be termed 
the "3 x 3 median filter" hereinafter. 

As shown in Fig. 3B, the target pixel is defined as a second 
one from the left edge and from the top edge in the input image 
5 1A. In this case, the local area is defined as a square area 1A2-2 
surrounded by a thick line in the image 1A while the target pixel 
is "black". The "3 x 3 median filter" is a filter that outputs the 
median value of the density values or gray-scale levels of the nine 
pixels (i.e., the pixel values) in the 3x3 local area 1A2-2. 
10 Since the median value is defined as a v^lue in the middle 

of a series of the nine pixel values arranged in order of magnitude 
of the density levels in the area 1A2-2, the median value is the 
density value of the white pixel, as shown in Fig. 3B. Therefore, 
the output of the extractor 11 for the target pixel of "black" in 
15 the local area 1A2-2 is "white". 

Next, for example, the target pixel is defined as a second 
one from the left edge and a third one from the top edge in the 
input image 1a. Then, the same filtering operation as above is 
carried out again. In this case, the output of the extractor 11 
20 fox the target pixel of "white" is "white". 

Similarly, the ^me filtering operation as above is 
repeated until every pixel of the input image 1A is defined as the 
target pixel . As a result, the intermediate image 11A shown in Fig. 
3A is formed. 
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As seen from Fig. 3A, in the intermediate image 11A, some 
"black" pixels at the oblique contour in the input image 1A are 
turned to "white" ones and at the same time, some "white" pixels 
at the same contour are turned to "black" ones. Such conversion 
between ""black" and ^white" does not occur in the remaining pixel? 
in. the image 11A. 

Compared with the input image 1A and the intermediate image 
11A, it is seen that both images 1A and llA includes j aggies at 
their oblique contours; however, the state or appearance of the 
j aggies is different from each other. This means that the colors 
of the pixels at the oblique contour in the image 11A are inverted 
with respect to those in the image 1A. Thus, the jaggy is inverted 
as well between the images 11A and 1A. 

Accordingly, when the input image 1A and the intermediate 
image llA are mixed together by the image miser 12 at the mixing 
ratio of 1 : 1, the inverted pixels in the images 11A and 1A are 
superposed together, thereby forming gray pixels in the output image 
3A ' as shown in Fig. 3A. in other words, since the input image 1A 
and the intermediate image 11A are averaged by the filtering 
operation of the image mixer 12, the inverted pixels in the images 
11A and 1A are suppressed or eased with each other. As a result, 
the oblique contour in the image 1A is smoothed, i.e., the jaggy 
in the image 1A is suppressed. 

Fig. 3C shows another example of the filtering operation 
-22- 
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of the extractor 11. in this example, a different input image IB 
from the input image 1A shown in Fig . 3A is processed by the processor 
2, where denotes an intermediate image out putted by the 

extractor 11 and "3B" denotes an output image. 
5 Compared with the intermediate images 11A and 11B of Figs. 

3A and 3C, it is seen that the image 11B includes no jaggy at its 
oblique contour. In other words, it is seen that the jaggy in the 
input image IB is eliminated in the intermediate image 11b. This 
is due to the magnitude or size difference of the jaggy between 
10 the imagas 11A and 11B. Therefore, in this case, the mixing ratio 
is set as 1 : o and the input image 1b is not mixed with the 
intermediate image 11B. Thus, the intermediate image 11B itself 
is outputted from the processor 2 as the output image 3B . 

As explained as above, with the image processing system 
15 according to the first embodiment of Fig. 2, the jaggy existing 
in the input image 1 can be eliminated or suppressed in the output 
image 3 by suitably setting the mixing ratio of the image mixer 
13 according to the input image 1 in the mixing ratio setter 13. 

Also, it is unnecessary to evaluate how the regions and/or 
20 objects are distributed in the input image 1 and to evaluate whether 
or not the input image 1 includes characters and/or lines are 
unnecessary. 

In the system of the first embodiment of Fig. 2, the 
functions of the median value extractor 11, the image mixer 12, 
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and the image mixing ratio setter 13 may be realized with a computer 
program to be prosecuted on a suitable computer system. In this 
case, the specific codes of the program may be recorded on any 
information-recordable medium, such as flexible disk (FD) , hard 
5 disk, magneto-optical disk (MO), CD-BOM, CD-R, magnetic tape, 
non-volatile memory card, and semiconductor Sead-Only Memory (ROM) . 
SECOND EMBODIMENT 
Although the median value extractor 11 that uses the median 
filter is provided in the image processing system according to the 
10 first embodiment of the invention to change the phase of the jaggy 
in the input image l r the invention is not limited to the use of 
the median filter. Any other filter that provides approximately 
-the same function as the median filter may be used. 

Fig. 4 shows an image processing system according to a 
15 second embodiment of the invention, which is used to carry out an 
image processing method according to the second embodiment. 

The image processing system of the second embodiment 
comprises an image processor 2A for processing an input image 1 
and for outputting the processed image as an output image 3. The 
20 image processor 2A comprises an average value calculator 14, a 
nearest value extractor 15, an image mixer 12, and a mixing ratio 
setter 13. 

The average value calculator 14 calculates the average 
value of the density or gray-scale levels of all the pixels {i.e.. 
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the pixel values) in a local area surrounding a target pixel of 
the input image 1 . 

The nearest value extractor 15 receives the input image 1 
and the output of the calculator 14 . Then, the extractor 15 provides 
a filtering operation that outputs the nearest value of the density 
or gray-scale levels of the pixel (i.e., the nearest pixel value} 
in the local area of the input image 1. The nearest pixel value 
thus obtained in the extractor 15 is sent to the image mixer 12. 
The mixer 12 performs its mixing operation by pixel by pixel. 

The image mixer 12 mixes the image subjected to the 
filtering operation in the extractor 15 and the input image 1 itself 
at a mixing ratio specified by the mixing ratio setter 13 , generating 
a mixed image. Then, the mixer 12 outputs the mixed image as the 
output image 3 . 

As explained as above, with the image processing system 
according to the second embodiment of Fig. 4, like the system of 
the first embodiment, the jaggy in the input image 1 can be 
eliminated or suppressed without evaluating how the regions and/or 
objects are distributed in the input image 1 nor whether or not 
the input image 1 includes characters and/or lines. 

As seen from the systems of the first and second embodiments, 
it is preferred to consider the contrast of the pixels in the input 
image 1 in the filtering operation. The effect to suppress the jaggy 
of the systems is more effective when the input image 1 has high 
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contrast, e,g., when the image 1 is an image created by computers, 
image scanners, or the like. Therefore, preferably, a means for 
evaluating the contrast of the input image I may be additionally 
provided. In this case, it is preferred that the image 1 is sent 
5 to the system of the first or second embodiment when the contrast 
of the image 1 is over a specific threshold value. 

In the system of the second embodiment of Fig. 4, the 
functions of the average value calculator 14, the nearest value 
extractor 15, the image mixer 12, and the image mixing ratio setter 
10 13 may be realized with a computer program to be prosecuted on a 
suitable computer, m this case, the specific codes of the program 
may be recorded on any information-recordable medium, examples of 
which are listed in the first embodiment. 

THIRD EMBODIMENT 

15 Fig. 5 shows an image processing system according to a third 

embodiment of the invention, which is used to carry out an image 
processing method according to the third embodiment. 

The image processing system of the third embodiment 
comprises an image processor 2B for processing an input image 1 

20 and for outputting the processed image as an output image 3 and 
an interpolation processor 5 for enlarging an original image 4 at 
a specific enlarging ratio through an interpolation process. The 
processor 5 is located outside the image processor 2B. 

The output of the interpolation processor 5 is used as the 
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input image 1 for the image processor 2, which is unlike the first 
and second embodiments explained above. 

The image processor 2B comprises a filter 16, an image mixer 
12, and a mixing ratio setter 13. 
5 The filter 16 provides a filtering operation that has been 

explained in the first or second embodiment for the input image 
1. The filter 16 may provide other filtering operation than those 
of the first and second embodiments if it outputs approximately 
the same result as shown in the first or second embodiment. 
10 " The image mixer 12 mixes the output of the filter 16 and 

the input image 1 (i.e., the output of the interpolation processor 
.5) together at a specific mixing ratio determined by the mixing 
ratio setter 13, The output of the mixer 12 or processor 2 is the 
output image 3 . 

15 As seen from Fig. 5, the system of the third embodiment is 

equivalent to one obtained by adding the interpolation processor 
5 to the configuration of the system of the first or second 
embodiment. In other words, the system of the third embodiment has 
substantially the same configuration as the system of the first 

20 or second embodiment except that the original image 4 is applied 
to the image processor 2B as the input image 1 after the image 4 
is enlarged or interpolated. 

As the enlargement process in the interpolation processor 
5, any interpolation method (e.g., the nearest neighboring or 
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bilinear method) may be used. As explained later, to utilise 
effectively the performance that j aggies can be eliminated or 
suppressed by the processor 2B, which varies dependent on the value 
of the enlarging ratio, the nearest neighboring method is suitable 
5 to this purpose. Needless to say, any other interpolation method 
may be adopted if it provides few blurs at the edges even if it 
generates j aggies at the contours. 

Next, the reason why the image processor 2B provides the 
output image 3 that preserves steep edges and suppresses contour 
10 j aggies is explained below with reference to Fig. 6. 

In Fig. 6, "MA" denotes an example of the original image 
4, which includes 25 (- 5 x 5) pixels. Like the first embodiment, 
the original image 4A includes only black and white pixels. "5A" 
denotes an example of the enlarged image produced by enlarging the 
■15 original image 4 A in the interpolation processor 5 r where the 
enlarging ratio is 1 : 2. In other words, the original image 4A 
is doubled to form the enlarged image 5A (I.e., the input image 
1) . The enlarged image 5A includes 100 (= 10 x. 10) pixels. The 
interpolation operation of the processor 5 for enlargement is 
20 carried out using the nearest neighboring method here, 

"16a" denotes an example of the filtered image produced by 
filtering the enlarged image 5A {i.e. , the input image 1) . The image 
16A is the output image of the filter 16 obtained by using the "3 
x 3 median filter", which has been already referred to in the first 
-28- 
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embodiment. 

The enlarged image 5A, which is the input image 1 to the 
image processor 2B, and the filtered image 16A (i.e., the output 
image of the filter 16) are mixed together in the image mixer 12 
5 at the mixing ratio of 1 ; 1. As a result, some of the pixels in 
the output image 3C are turned from black or white to gray near 
the edges or contour. 

As seen from Fig. 6, the enlarged image 5A includes steep 
edges; however it includes conspicuous oblique jaggy. Compared 
^ 10 with the image 5A with the output image 3C, it is seen, that the 
fn conspicuous oblique jaggy in the image 5A disappears in the image 

yj 3C. This means that the oblique jaggy in the image 5A is eliminated 

III by mixing the images 5A and 16A with the image mixer 12 at the mixing 

O ratio of 1 : l. Moreover, the steep edges at the horizontal and 

til 

= 15 vertical contours in the image 5A are kept substantially unchanged 
H in the output image 3C, which has been difficult to be realized 

by the conventional interpolation methods. 

With the image processing system according to the third 
embodiment of Fig. 5, as explained above, there is an advantage 
20 that the output image 3 includes almost no jaggy and steep edges 
substantially unchanged. 

In the system of the third embodiment, the enlarging ratio 
of the original image 4 may be set optionally as necessary . However, 
it is preferred that the enlargement ratio of the original image 

-29- 
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4A is equal to an integer. In particular, it is most preferred that 
the enlargement ratio of the original image 4A is equal to two, 
S5 shown in Fig, 6. This is because the elimination of the jaggy 
by the processor 2B is performed most effectively in this case. 
5 The system of the third embodiment is suitable when the 

original image 4 is an image including conspicuous j aggies - 
Especially, it is mora suitable to artificial digital images for 
personal computers (PCs) including texts and/or drawings.. 
Needless to say, it may be applicable to any image including natural 
10 pictures. 

In the system of the third embodiment of Fig. 5, the 
functions of the interpolation processor 5, the filter 16, the image 
mixer 12, and the image mixing ratio setter 13 may be realized with 
a computer program, to be prosecuted on a suitable computer. In this 
15 case, the specific codes of the program may be recorded on any 
informat ion-recordable medium, examples of which are listed in the 
first embodiment. 

FOURTH EMBODIMENT 
Fig. 7 shows an image processing system according to a 
20 fourth embodiment of the invention, which is used to carry out an 
image processing method according to the fourth embodiment. This 
system has the same configuration as the system of the third 
embodiment of Fig. 5 except that the mixing ratio ' of the mixing 
ratio setter 13 is determined on the basis of the information 
-30- 



'01 05/24 21:10 FAX 03 3208 0844 



ARICHIM&IZUII 



BLUMENTHAL ( FOLEY |036 



obtained in the interpolation processor 5. 

fig-. 8 shows the process of the system of the fourth 
embodiment, in which an example of the original image 4 is enlarged 
to form an enlarged image 5B at the enlarging ratio of 1.6 by the 
5 interpolation processor 5 using the nearest neighbor method. As 
seen from Fig. 8, since the enlarging jratio is not equal to an integer, 
each of the pixels of the original image 4A can be assigned to 2 
x 2 (=4) pixels, 1x2 (=2) pixels, 2x1 (=2) pixels, or 1 x 
1 (= 1) pixels of the enlarged image 5B. Generally, there are four 
10 assignment ways of each pixel of the original image 4A to the pixel 
or pixels of the enlarged image 5B when the enlarging ratio is equal 
to or less than two. 

It is seen from Fig. 8 that the pixels of the original image 

4A assigned to the 2x2 (=4) pixels of the enlarged image 5B include 

111 

15 a jaggy in their neighborhood. Thus, it is sufficient that the 
!~= operation to eliminate or suppress jaggies is applied to the pixels 

of the image 4A assigned to the 2x2 (=4) pixels of the image 
5B. 

A filtered image 16b in Fig. 8 is produced by the filtering 
20 operation of the filter 16 to the enlarged image 5B, where the "3 
x 3 median filter" is used. As seen from comparison between the 
enlarged images 16A and 16B, the line with a width of one pixel, 
which is formed by the pixels l6Ba, disappears in the image 16B. 
Thus, to overcome this problem, the assignment information is sent 
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from the interpolation processor 5 to the mixing ratio setter 13, 
The setter 13 change? or adjusts the value of the mixing ratio 
according to the assignment information thus sent. 

For example, when the "2 x 2 assignment" is made, i.e., the 
5 pixel of the original image 4R is assigned to the 2x2 (=4) pixels 
of the enlarged image 5B, the mixing ratio is set as 1 ; 1. This 
means that the filtered image 16B and the enlarged image 53 (i.e., 
the input image 1) are mixed or added together at 1 : 1 (e.g., 5 : 
5) , thereby forming an example 3D of the output image, as shown 
10 in Fig. 8. In this case, the pixel 3Da of the output image 3D has 
the same density value as the pixel 5Ba of the enlarged image 5B, 
Unlike this, when any other assignment then the "2 x 2 
Hi assignment" is made, i.e., the pixel of the original image 4A is 

assigned to the 1x2 (=2) pixels, 2x1 (=2) pixels, or 1 x 1 
15 (= 1) pixel of the enlarged image 5B, the mixing ratio is set as 
0 : 1 (e.g., 0 : 10). This means that only the enlarged image 5B 
(i.e., the input image 1) is outputted from the image mixer 12 as 
the output image 3, thereby forming the image 3D shown in Fig. 8. 
In this case, the pixel 3Da of the output image 3D has a density 
2 0 value equal to half the density values of the pixel 16Bb of the 
filtered image 16B (i.e., gray). 

As seen from the output image 3d in -Fig. 3, the jaggy at 
the oblique contour is eased or eliminated while the edges are kept 
steep . 
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As described above, with the image processing system 
according to the fourth embodiment of Fig. 7, the interpolation 
processor 5 sends the assignment information to the mixing ratio 
setter 13 and then, the setter 13 changes or adjusts the value of 
5 the mixing ratio according to the information thus seat- Therefore, 
the enlarged image 5B having less quality degradation can be 
produced with the interpolation processor 5 not only when the 
enlargement ratio of the original image 4& is equal to an integer 
but also it is unequal to an integer. 

10 The nearest neighbor method is used in the interpolation 

processor 5 as the interpolation function here. However/ any other 
interpolation method may be used for this purpose if it provides 
suitable enlargement of the original image 4A. 

For example, an interpolation function obtained by 

15 combining the nearest neighbor method with the bilinear method may 
be used. When the inter-pixel distance of the original image 4A 
is defined as 1, the, interpolation function used in the nearest 
neighbor method is given by the line shown in Fig. The 
Interpolation function used in the bilinear method is given by the 

20 line shown in Fig. 9B. 

As explained earlier in this specification, the nearest 
neighbor method preserves the steep edges of the regions and/or 
object? in an image while the image quality degrades due to j aggies 
existing at the contours of the regions and/or objects. The 
-33- 
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bilinear interpolation method suppresses the j aggies occurring in 
the nearest neighbor method while the image quality degrades because 
the edges of the regions and/or objects in the image tend to blur. 
Thus, each of the interpolation functions of Figs. 9A and 9B has 
5 its merits and demerits. To overcome this trade-off simply, the 
inventor created the interpolation function, as shown in Fig. 9C. 
This function corresponds to the combination of the functions of 
Figs. 9A and 9B. 

In the function of Fig. 9C, a is a constant greater than 
10 0.5 and less than 1 (i.e., 0.5 < a < 1) . Optional value change of 
the constant a. causes the output image 3. When the value of a is 
set as small within the range of 0.5 < a. < 1, the edges of the image 

3 will be sharp. On the contrary, when the value of a is set as 
large within the range of 0.5 < a < 1, the total balance of the 

15 imag<a 3 will be good. 

It is preferred that a suitable one of the interpolation 
functions is selected according to the type of the original image 

4 and/or the enlarging ratio of the interpolation processor 5. 
Needless to say, the cubic convolution method may be used for this 

20. purpose. 

with the system according to the fourth embodiment of Fig. 
7, there is an advantage that the output image 3 includes no j aggies 
and steep edges even if the original image 4 is enlarged at an 
optional enlarging ratio (in particular, the ratio is unequal to 
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an integer) in the interpolation processor 5. 

In the image processing system of the fourth embodiment of 
Fig. 7, the functions of the interpolation processor 5, the filter 
16 , the image mixer 12, and the image mixing ratio setter 13 may 
5 be realized with a computer program to be prosecuted on a suitable 
computer. In this case, the specific codes of the program may be 
recorded on any information-recordable medium, examples of which 
are listed in the first embodiment. 

FIFTH EMBODIMENT 

'% 10 Fig* 10 shows the configuration of a display controlling 

IB 

fjl apparatus 37 according to a fifth embodiment of the invention, which 

y comprises an input buffer 27, a frame buffer 28, and an output buffer 

III 29 along with an interpolation processor 25 and an image processor 

W 22. An image or picture is inputted into the apparatus 37 from an 

^ 15 image /picture source 26. The output of the apparatus 37 is sent 
to a display device 30. The device 30 has a raster scanning function 
and therefore, the image or picture image/picture sent from the 
source 26 is prepared for the same function. 

As the interpolation processor 25 r one of the interpolation 
20 processors 5 used in the first to fourth embodiments may be used. 
As the image processor 22, one of the image processors 2 used in 
the first to fourth embodiments may be used. 

An original image and/or picture 4 sent from the source 2 6 
is temporarily stored in the input buffer 27 for buffering. The 
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original image 4 in the buffer 27 is then sent to the interpolation 
processor 25, in which the image 4 is subjected to a specific 
interpolation operation, thereby enlarging the image 4, The output 
of the processor 25 (i.e., the enlarged image) is then sent to the 
5 image processor 22, in which the jaggies in the enlarged image are 
eliminated. The output of the processor 22 (i.e. r the filtered, 
enlarged image) is temporarily stored in the output buffer 29 and 
then, it is sent to the display device 30. Thus, the original image 
4 is displayed on the screen of the device 30 at an enlarged or 

10 interpolated resolution. 

As the display device 30, a CRT, LCD, PDP f or EL display 
may be used. However, it is preferred that the device 30 is of the 
type having a fixed resolution, such as a LCD, PDP, and EL display, 
because they are of the dot-matrix type where the pixels are arranged 

15 in a matrix array and they have a fixed resolution. 

With the display 'controlling apparatus 37 according to the 
fifth embodiment of Fig. 10, the output image 3 including no j aggies 
and steep edges can be displayed on the screen of the display device 
30. 

20 VARIATIONS 

Needless to say, the invention is not limited to the 
above- described first to third embodiments. For example, the 
invention may be applied to a system comprising a plurality of 
apparatuses or devices, or to a single apparatus or device. 
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While the preferred forms of the present invention have been 
described, it is to be understood that modifications will be 
apparent to those skilled in the art without departing from the 
spirit of the invention. The scope of the present invention, 
5 therefore, is to be determined solely by the following claims. 



